Taking the two questions you flagged for challenge, in the order I can actually help with.

1. Task shape: our grid varied shape and apparatus separately, and it supports your inference

You wrote that an all-coding benchmark never varied shape, so it measures apparatus rather than shape, and asked for that to be confirmed or shot down. Our C0 to C9 grid crossed four task shapes against ten apparatus levels, 40 samples per condition, so it can separate them.

Shape is real and it is large. Firing rate across the whole grid, all conditions pooled:

Summarization never fired once in any condition, including with no system prompt at all. That is a floor no amount of apparatus explains.

But code specifically is a conjunction, not a shape effect:

Code fires fully with no apparatus, collapses to zero under a bare named persona, and comes back to 10/10 once the full agent prompt is present. So "coding-shaped tasks suppress" and "coding-shaped tasks reason most" can both be true readings of the same model depending on which single apparatus level you happened to hold fixed.

Which is your inference, confirmed: a benchmark that holds shape constant at coding and carries no apparatus is sitting at our C0 cell, where code fires 10/10. It is measuring the apparatus axis at one point, not the shape axis. It cannot see the C4 collapse and it cannot see the summarization floor.

Raw: gate-study/logs/grid_turns.jsonl.

2. On the sampling confound in the +2.64

You already caught that thinking-on ran t0.7 and thinking-off t0.6. I would add that the direction of that confound is not obviously benign for the claim. On a verifiable-answer codegen benchmark the lower-temperature arm is usually the one you would expect to score higher, so a lower-temperature arm losing by 2.64 points is not the shape you would predict from temperature alone. That argues the effect is probably real rather than manufactured, but it also means the magnitude is not clean, and +2.64 is small enough that a two-variable comparison cannot really defend it as a number. Scope-limited merge with the magnitude marked provisional seems right to me, if replication does not arrive.

3. On cap-hits as failures rather than truncations

This one I can support from a different angle, and it generalizes past Laguna.

We ran a six-requirement acceptance-criteria coding task on Qwen 3.6 35B-A3B at a fixed 4096 ceiling, thirty times across three prompt conditions. It returned completely empty content, having spent the entire budget inside the reasoning block, in 28 of 30 runs. Laguna on the same task and ceiling did it 9 times in 30. Not wrong answers, no answer at all.

That is the same failure your §5f describes, on a different model and a different serving stack, which I think promotes "treat cap-hits as failures" from a Laguna harness note to a general benchmarking rule. It also matches the degeneration-loop reading rather than the needs-more-budget reading: 4096 is not a stingy ceiling for that task, and more of it was not going to help.

Raw: cross-model/logs/criteria_turns.jsonl.

4. On the regime split being real versus being the benchmark

I do not think our data settles this and I would not claim it does. What I can say is that the two regimes differ on more than task shape: single-turn codegen with a verifiable answer has no apparatus and no accumulated context, and your behavioral and long-agentic arms have both. Since apparatus is now an established axis with a large effect, "regime" and "apparatus dose" are currently confounded with each other in the comparison. Splitting them would need the codegen benchmark re-run with a full agent prompt attached, which is a different experiment from the one anybody has run.

Worth stating in the section if it lands, because otherwise the regime table reads as a property of task type when it may partly be a property of how much scaffolding each regime happens to carry.

5. Two small things

Nothing to do with the claim, just noticed while reading the diff.

• The card's footer still says three independent stacks while the validation row above it now says four.
• Quickstart row 1 and the §5f table are consistent, but Quickstart row 4's ~12k ceiling and §5g's advice on truncated-think interact: a 12k cap makes truncated-think turns more likely than a 16k one did, so the row 4 change slightly raises exposure to the §5g failure unless the placeholder-substitution fix is in place. Might be worth one clause pointing row 4 at §5g.

On replication

It is running now, on full-precision Laguna S 2.1 NVFP4 at rev 0761412, vLLM. HumanEval+, both arms, with the two things your review asked for built in:

• Temperature held identical across arms. That is the whole point of running it: the +2.64 currently rests on a two-variable comparison, and this arm removes the sampling half so what is left is the enable_thinking half on its own.
• A 12k ceiling, which also puts @apollo-mg's own p95-derived recommendation under test rather than just adopting it.
• Per-sample cap-hit and extractable-answer logging, so the degeneration-loop claim can be checked directly rather than inferred: if cap-hitters really are loops rather than truncations, that shows up per sample, and the 11-of-492 no-extractable-answer figure gets an independent comparison point.

Results to this thread when they land, expected within a day, either way. If it comes back with ON ahead on our stack too, that is the replication you asked for. If it comes back flat or reversed once temperature is controlled, that is worth knowing before this changes Quickstart #1, and I will say so just as plainly. Raw data published the way everything else in our lab repo is.

Correcting myself on one of the two small things above, before you spend time on it.

I said the card's footer still says three independent stacks while the row above says four. That is wrong, and it is my misreading rather than a bug in your diff. The two numbers count different sets: the header counts every stack including your own, the footer counts the ones independent of you. Main is "three stacks" in the header against "2 independent stacks" in the footer, and this PR moves both together to four and 3. Same convention, correctly updated. Nothing to fix.

The second bullet still stands as far as I can tell: a 12k ceiling makes truncated-think turns more likely than 16k did, so Quickstart row 4 and the §5g failure interact, and a pointer between them would be worth one clause.

Apologies for the noise on the first one.

will fix the two small things thanks @Blackwellboy

Thanks, this is the most useful review comment the repo has had. Taking your points in order, then what I am running.

1. Task shape: inference confirmed, and I had just talked myself out of it

Your pooled table is the thing I did not have. I re-derived it from grid_turns.jsonl before writing anything, and it matches exactly:

Awkward timing on my end and worth stating plainly: about an hour before your comment landed I shipped a commit that declared this inference wrong and retired it, on the strength of your C0-vs-C4 code split alone. That was an over-correction, it stood for exactly one commit, and it is fixed in #12 with the retraction visible rather than quietly reverted. The two parts do not compete: shape is a large independent effect (the summarization floor is a floor no apparatus explains) and code specifically is a persona conjunction. A shape-constant no-apparatus benchmark sits at C0 where code fires 10/10, so it cannot see the C4 collapse or the summarization floor. That is your confirmation, folded.

Everything from this comment that touches §2 is in #12, not here, since #12 is the section rewrite. Your review on that one is welcome, particularly on whether "task and persona interact" is more weight than one grid should carry.

2. The +2.64 and the direction of the confound

Your framing is better than mine and I have adopted it: on a verifiable-answer codegen benchmark the lower-temperature arm is normally the one you would expect to score higher, so a lower-temperature arm losing by 2.64 is not the shape temperature alone predicts. That argues the effect is real while the magnitude stays unclean. Agreed on scope-limited merge with the magnitude marked provisional if replication does not land.

3. Cap-hits as failures: promoted to a cross-model pattern

I verified both lanes per-sample from your published logs rather than from your table: cross-model/logs/criteria_turns.jsonl gives 28/30 empty-content and gate-study/logs/criteria_turns.jsonl gives 9/30. That is now a standalone entry in patterns.md on main, "An empty response at a token cap is a failure, not a truncation," with the two harness rules (score cap-hits as failures; log the rate per arm, because dropping them inflates whichever arm degenerates most, normally the thinking-on one). Cited to your runs.

4. Regime and apparatus are confounded

This is the sharpest point in your comment and I had not seen it. Single-turn codegen with a verifiable answer carries no apparatus and no accumulated context; our behavioral and long-agentic arms carry both. Since apparatus is now an established large-effect axis, "regime" and "apparatus dose" are confounded with each other in the very comparison this PR is about, so the regime table may partly be measuring scaffolding rather than task type. It will be stated in the section if this lands. Separating them needs the codegen benchmark re-run with a full agent prompt attached, which nobody has run, and I am noting it as the follow-up experiment rather than pretending this PR settles it.

5. The two small things

Noted on the footer, no apologies needed, and your own correction is right about the convention. Unrelated: main's footer now reads "2 independent replications" instead of "2 independent stacks", same convention, just less ambiguous.

The second bullet is fixed, in b199529 on this branch. You had the section letter off by one (§5f is truncated-think, §5g is the gfx1151 serving config) but the interaction you identified is real and I had the two recommendations sitting in the same row with no pointer between them. The advice is now split by regime, because 12k is a p95 of single-turn outputs:

• Single-turn / benchmark: ~12k, score cap-hits as failures.
• Agent loop: keep headroom, and separately make the loop survive a capped turn rather than relying on the cap never being hit, since the failure is silent and kills every subsequent turn.

What I am running now, on a third stack

Since you are covering NVFP4/vLLM, I have put ours on the other side of the serving axis so the two runs are not the same experiment twice: Q4_K_M GGUF on poolside's own llama.cpp fork at 04b2b72cb, GB10, -fa on --jinja --no-mmap, 4 slots. HumanEval+ 164 problems, both arms, scored by the official evalplus scorer rather than anything I wrote.

Controls, matching what your review and mine both asked for:

• temperature identical across arms (0.6/0.95 both), so the sampling half is removed and what is left is the enable_thinking half
• max_tokens identical at 12,288, which puts @apollo-mg's own p95-derived ceiling under test rather than adopting it
• arms interleaved per problem, not blocked
• cache_prompt: false plus a per-request nonce
• no system message at all, deliberately, to keep off the persona axis your grid just established as large
• per-sample cap-hit, extractable-answer, and reasoning unique-line-ratio logging, so degeneration-loop-vs-truncation is checkable rather than inferred

Then a follow-up pass re-running only the cap-hitters at a raised ceiling, which is the direct test of "raising the budget does not convert cap-hitters into passes." If they convert, that claim is wrong on this stack and I will say so.

Results and raw JSONL to this thread either way, including if it comes back flat or reversed. Two independent replications on two different quants and two different serving stacks is a much better basis for changing Quickstart #1 than one screenshot was.

Note for whenever this does merge: this branch predates today's §2 rewrite and the patterns additions, so it needs a rebase on main first. Its card still carries the retracted dose-curve row.

Glad the pooled table was the missing piece, and respect for leaving the over-correction visible instead of quietly reverting it. That is the standard this repo keeps setting.

Your llama.cpp run design looks right to me, and the deliberate no-system-message call is exactly the control that matters now. Coordination point so our two runs stay maximally comparable: mine is also no system message (deliberately the C0 cell), temperature fixed identically across arms, 12,288 ceiling, arms interleaved with nonces, all 164 HumanEval+ problems, evalplus scoring, per-sample cap-hit and extractable-answer logging. So the only intended differences between our runs are quant and serving stack, NVFP4/vLLM against Q4_K_M/llama.cpp, which is what makes the pair informative. If yours and mine land the same sign on opposite stacks, that is about as settled as this question gets short of Poolside's own data. Results to this thread as soon as my run completes, raw JSONL alongside as usual.

Two small notes on the cap-hit pattern entry, since I re-derived it from my raw before saying anything.

The numbers are all clean: 28/30 and 9/30 both reproduce, both really are three prompt conditions per model, and every one of those empty responses really did stop at exactly 4,096 completion tokens. Nothing to fix there.

One sentence overreaches slightly though. The entry says "the same request under a lower apparatus dose completes fine, which is what separates a loop from a genuine truncation." That holds for Laguna and not for Qwen. Per condition:

Laguna behaves the way the sentence describes, fine at low dose and collapsing under the agent prompt. Qwen returns nothing at every dose including the bare prompt. If anything that strengthens the entry's conclusion rather than weakening it, because a failure that does not depend on apparatus at all is even harder to read as a truncation, but the stated discriminator is Laguna-only and I would not want it carried as a general rule. Raw: gate-study/logs/criteria_turns.jsonl and cross-model/logs/criteria_turns.jsonl.

Also minor, you described that entry as being on main. It is currently in #12 rather than main, so it will land when that merges. Flagging only so it does not get lost if #12 gets reshaped.

Interim from our replication, 77/328 requests in

Third stack, other side of the serving axis from @Blackwellboy's NVFP4/vLLM run: Laguna S 2.1 Q4_K_M GGUF on poolside's own llama.cpp fork at 04b2b72cb, GB10, -fa on --jinja --no-mmap, 4 slots. HumanEval+ 164 problems x 2 arms.

No pass-rate numbers in this comment. Scoring is done by the official evalplus.evaluate at the end, not incrementally, and I am not going to eyeball a partial pass@1 and then have to walk it back. What follows is only what the run measures directly.

Paired tasks with both arms complete (n=38):

Four things worth putting on the record now, because two of them bear on the methodology half of this PR rather than on the +2.64.

1. enable_thinking: false is structurally off on a third stack. 0 of 38 fired, 0 of 61 counting unpaired rows. That is an independent confirmation of @Defilan's render-check finding on a different quant and a different fork, and it makes this the strongest-evidenced claim in the guide. It also means the arms here really are separated on the variable of interest.

2. Zero cap-hits at the 12,288 ceiling, so far. Not one request in 77 has hit it. If that holds to 328, the cap-retry pass I built has nothing to re-run, and the honest reading is not "the claim is wrong" but "the ceiling is not the binding constraint on this stack", which is a different and weaker statement than the guide currently makes.

3. Our p95 is roughly a third of theirs, which is the interesting divergence. @apollo-mg reported p95 10,152 and median 1,945, and derived ~12k from it. We are measuring p95 3,093, median 1,069, longest single response 5,025. Same benchmark, same task shape, no system message in either case. So the ~12k figure looks like a property of their stack and sampling rather than of the model, and a guide-level "use ~12k" recommendation sourced from one stack's p95 is doing more extrapolating than the data supports. Worth reconciling against @Blackwellboy's run when it lands, since he is also using 12k.

4. The cost gap is large and has to be part of the verdict. ON is spending 7.7x the output tokens (1,069 vs 138 median) and 3.8x the wall clock (147s vs 39s). Whatever pass-rate delta comes out, that is the price it has to beat, and a +2.6 point gain bought at 7.7x tokens is a different recommendation from a +2.6 point gain that is free. The current PR text does not price it at all, and it should.

Incidental cross-check on @Blackwellboy's grid, free from this run: 164 bare code-shaped prompts with no system message and thinking on is the same cell as his C0 code, which fired 10/10. We are at 34/38 (89%) and the four that did not fire are HumanEval/3, /8, /9, /15. So code firing is high but not saturated here, in the same direction as the bare-prompt gap already noted in the guide (his 75% against our ~50%).

Timeline: 389 request-minutes of work left at 4 concurrent slots, so ~1.6 hours to all 328 rows, then official evalplus scoring on both arms plus the cap-retry pass if anything hit the ceiling. Numbers and raw JSONL to this thread after that, including if it comes back flat or reversed.

I am deliberately not raising parallelism to speed this up: the model is 71 GB resident on a 121 GB unified-memory box, and doubling the KV allocation to get more slots is how that box starts swapping and stops being a valid measurement.

Figured I'd jump in with what I've got so far.

Laguna-S-2.1 test data — Apollo fleet

Raw per-sample data behind the numbers quoted in #10.

Hardware (all runs): quad Tesla P100-PCIE-16GB (sm_60), 1063 MHz / 150 W, single node.
Server: poolside llama.cpp fork, -c 32768 -np 1 -ngl 99 -sm layer -ts 1,1,1,1 -fit off
Model: Laguna-S-2.1-UD-Q2_K_XL.gguf — Q2_K_XL. This matters; see CONFOUNDS.
Benchmark: HumanEval+ (EvalPlus extended tests), full 164 problems.
Dataset identity is fingerprint-verifiable with humaneval-plus/fetch_dataset.py
(164 problems, 10,730,825 test bytes, HumanEval/0 test = 77,039 chars).
max_tokens: 16000 on every run.

1. Thinking ON vs OFF — HumanEval+ full 164, K=3 (492 samples per arm)

Each arm at its own card-recommended sampling.

Files: humaneval-plus/laguna_hep_results_t07_k3.json (ON),
laguna_hep_results_t06_k3_nothink.json (OFF). Per-problem buckets, finishes,
srcs, out_toks, rc_chars.

Thinking-ON is inferred, not directly measured, for the t0.7 arm. That run predates the
harness's enable_thinking plumbing, so its metadata carries no thinking flag and no
reasoning_content lengths. Supporting evidence: token medians ON > OFF on 164/164
problems, median ratio 6.51×, 0/492 ON samples at or below the OFF median.

2. Stopping-rule observation

WRONG counts are near-identical between Laguna-ON (30) and a much larger model run on the
same node, same harness, same 492-sample design (Puzzle-75B-A9B-UD-IQ4-XL, 28 WRONG,
93.90 % pass@1 — humaneval-plus/puzzle_hep_results_t10_k3.json).

Laguna produced no extractable code on 11/492 samples; Puzzle on 1/492.

Conditioning on samples that produced code, the 3.05-point gap narrows to 1.16 pt
(EXEC_TIMEOUT counted as wrong code) or 0.58 pt (counted as non-answer). Both
definitions give the same direction.

Counter-evidence against over-reading it: forcing termination via enable_thinking: false
removed all 11 non-answers but moved WRONG 30 → 56, a net loss of 2.64 points.

3. Persona × tools factorial

15 HumanEval+ problems (every 10th), K=1, t0.7 / top_p 0.95 / top_k 20, thinking at
template default. Each cell in its own subshell so env could not leak.

Persona string: "You are a senior software engineer. Write clean, correct, production-quality Python."
Tools: 3 schemas (read_file, write_file, run_command) on every request, never called back —
no tool results re-entered context.

Independent composition would predict 1.09 × 0.92 ≈ 1.00×; observed 0.39×.

Robustness (n=15, so this matters):

The persona row's ratio of means is 1.51×, which reads as "+51 % reasoning." That is one
problem: HumanEval/30, 2 260 → 71 375 chars. Drop it and the ratio is 0.96×. The mean
was reported first and had to be retracted.

Files: persona-x-tools/sup_results_{base,persona,tools,both}.json,
analysis in cell_pairwise.py.

CONFOUNDS AND SCOPE LIMITS

1. Q2_K_XL only. The 2-bit tax is entangled with every result here, and specifically
   with the wedging. A Q4_K_M run reporting a much lower token ceiling and zero cap-hits
   would be consistent with the wedging being a quant artifact rather than a Laguna
   property. This has not been tested here.
2. "Cap-hitters are degeneration loops" is not established by this data. That claim
   comes from an earlier temp-0 K=1 leg via compression-ratio stratification (loops
   < 0.08, coherent-capped > 0.12). An attempt to validate that threshold on a second model
   failed to test it — the model never wedged, so there were no positives. Treat as
   inferred.
3. Single benchmark, single turn. HumanEval+ is single-turn code generation. No
   long-agentic-loop data here at all.
4. The 2×2 is K=1 over 15 problems. It sizes an effect; it does not establish a rate.
   Firing failed only in the both cell, 2/15 — Fisher p ≈ 0.48 against 0/15, i.e. not
   statistically distinguishable. The defensible claim is the length collapse, not a
   firing-rate claim.
5. The persona bundles an identity claim with a quality demand. Which half drives the
   interaction is unseparated.
6. Tools were passed but never called back. A real agent loop feeds tool results into
   context every turn; that is a further untested and plausibly stronger variable.
7. Server flags for the two eval legs were not recorded (they predate the launcher script).
   -fa on/off was never tested on Laguna.
8. The 15 problems in the 2×2 are every 10th HumanEval+ id, not a random sample.

Files

humaneval-plus/
  laguna_hep_results_t07_k3.json      thinking ON, per-problem
  laguna_hep_results_t06_k3_nothink.json  thinking OFF, per-problem
  puzzle_hep_results_t10_k3.json      larger-model comparison, same harness/node
  *.log                               console output for each
  hep_eval.py                         harness (revision as of 2026-07-25)
  fetch_dataset.py                    rebuilds + fingerprint-verifies the dataset
  SUMMARY.md                          full writeup incl. method notes
persona-x-tools/
  sup_results_{base,persona,tools,both}.json
  sup_{base,persona,tools,both}.log
  cell_pairwise.py                    paired/sign-test analysis
  SUMMARY.md                          full writeup

laguna_apollo_data_20260726.tar.gz

This is exactly what the thread needed, thanks for shipping the raw. The Q2_K_XL disclosure is the big one: it means the three replications now span the quant axis by accident, yours at 2 bit, Tom's at Q4, mine at NVFP4, all on HumanEval+ both arms. If the sign survives all three, that's a model property. If it fades with quant quality, that's its own finding. Mine also logs thinking firing directly per sample (the plumbing yours predates), so we'll get a measured rather than inferred ON arm at the high end. Results shortly, the run is nearly complete.

One note from adjacent data: your caution on cap-hitters as degeneration is warranted. On Qwen the identical symptom turned out to be pure truncation, budget converts 8 of 10 empties to valid at 8192 with zero degeneration in any tail. Different model, but it says the symptom has more than one cause and per-model checking is right.

Shared your response with my agent, got a test queued up.

Opus 5: HumanEval/47 hit the 16 k ceiling twice and passed both times — extractable, correct code, then it kept generating until the cap. So finish_reason=length isn't a failure signal on its own, and my config-doc line saying cap-hits must be bucketed as failures was wrong. Fixed, with the correction visible rather than reworded away.

The full breakdown of the 12 cap-hits:

│ cap-hit │ bucket │ problem pass_frac │
│ HumanEval/47 ×2 │ PASS │ 1.00 │
│ /44, /90 │ TRUNCATED │ 0.67 │
│ /118 ×2 │ TRUNCATED │ 0.33 │
│ /76, /116 ×2, /132, /145 ×2 │ TRUNCATED │ 0.00 │

6 of the 10 truncations sit on problems Laguna never solves in any sample — which is consistent with "budget won't help" but doesn't demonstrate it, because a never-solved problem is also never solved with more budget. Both hypotheses predict the same thing there. The 4 truncations on solvable problems — /44, /90, /118×2 — are the only place the hypotheses diverge, and that's the population BlackwellBoy's test has to target.

The decisive experiment is now specified and ready: re-run those 8 problems at a much larger budget and watch whether the 4 solvable ones convert. One wrinkle worth knowing — our 16 k ceiling wasn't a policy choice, it's the server's -c 32768, so testing this needs Laguna relaunched with a bigger context, not just a bigger max_tokens. If truncations convert, our stopping-rule recommendation is wrong for this model and the fix is simply more budget. If they cap again at 48 k, degeneration is confirmed.

It's queued behind stage 2 on .194, same as the loop detector — and it's the better experiment of the two, since it has guaranteed positives where the detector test didn't.

Worth noting what BlackwellBoy set up by accident: the three replications now span 2-bit, Q4_K_M, and NVFP4 on the same benchmark and both arms. If our +2.64 survives at NVFP4 with directly-measured firing, it's a model property; if it fades as quant quality rises, that's a cleaner finding than the original. Either outcome is worth more than our single point, and his run gets the measured ON arm that ours could only infer.

Replication results, as promised. Full-precision Laguna S 2.1 NVFP4, rev 0761412, vLLM 0.25.1, HumanEval+ all 164 problems, both arms, temperature identical.

Verdict first

The +2.64 does not replicate once the temperature confound is removed. On HumanEval+ — the metric the claim was made on — thinking ON scored 89.84% vs OFF 90.85% (mean of 3 seeds; ON −1.02). Paired per problem it is simply flat: 10 problems favor ON, 13 favor OFF, 141 tied. On base HumanEval the delta leans the other way (ON +1.22). Both deltas are inside seed noise, and both are smaller than the gap between scoring variants of the same benchmark within one run.

So on a second stack, with one variable instead of two, the accuracy half of the single-turn-codegen row is flat-to-reversed, not +2.6. The temperature difference appears to have been the whole accuracy story — consistent with your own review note that a lower-temperature arm losing by 2.64 was not the shape temperature alone would predict.

What DOES replicate — both of the methodology claims

1. Flakiness: direction confirmed. ON 11 flaky problems vs OFF 17 (theirs: 11 vs 24). Not halved here, but ON is genuinely more run-to-run stable. This is the defensible part of the regime row.
2. Cap-hits are failures, not truncations: strongly confirmed. 15/492 ON runs hit the 12,288 ceiling; 14 of them returned zero extractable code, whole budget inside the think block, tail compression ratios 2.9–143× (coherent text ~2.5–3×; 44/133/143 are hard loops). ON's p95 completion is 6,763 tokens — cap-hitters are not the tail of the length distribution, they are a separate failure mode more budget cannot fix. The loops concentrate on specific problems (HumanEval/116 and /132 looped in all 3 ON seeds). One OFF run also looped (40k chars of content, ratio 20) — same failure mode, 1/15th the rate.
3. ~12k ceiling: adequate on this stack. Nothing non-looping came near it; 12k vs 16k loses nothing and caps the cost of each loop.
4. No-extractable-answer: ON 22/492 (~4.5%: the 14 loops + 8 finish=stop responses with no fenced code) vs OFF 1/492. Their 11/492 is the right order of magnitude for ON. A loop-detection stopping rule would recover at most ~1 pt for ON here — i.e. it brings ON to parity on HumanEval+, not ahead.

Design (the confound eliminations your review asked for)

• Temperature identical both arms: t 0.7, top_p 0.95, top_k 20 (model-card sampling), sent explicitly per request. 3 seeds per (problem, arm), same seed both arms of a pair; pass@1 = mean ± sd.
• No system prompt, user message only — deliberately the C0 cell from our grid, which is where an all-coding no-apparatus benchmark sits. With the kwarg on, thinking fired 492/492 (C0 code = 10/10 reproduces); control arm fired 0/492.
• Arms explicit enable_thinking: true|false; byte-identical HumanEval+ prompts (no nonce — per-request vLLM seeds instead; vLLM prefix caching is exact-KV reuse, so the llama.cpp cached-reply hazard the nonce guards against doesn't exist here); arms interleaved per problem; max_tokens 12,288 fixed, no retry; scoring = evalplus 0.3.1 sanitize + its own test execution, no LLM in the scoring path; 984 requests, 0 errors.

Numbers

That last column is worth a sentence: on this stack thinking ON costs ~11× wall clock for a flat accuracy result. Whatever the regime table ends up saying, "ON for single-turn codegen" is not free.

What I'd suggest for the merge decision

• Land §5f's methodology findings (cap-hits-are-failures, ~12k ceiling, loop-detection rule) — two-stack confirmed now, and the loop-degeneration mechanism generalizes (we've seen the same signature on Qwen 3.6 at a 4096 ceiling, 28/30).
• Do not change Quickstart Laguna S 2.1: make the guide answer "what should I do", + @BlackwellBoy's 12h soak #1 to "true for single-turn codegen." The accuracy half of that row didn't survive temperature control on a second stack. If the row lands at all, scope it to stability: "ON reduces run-to-run flakiness on verifiable codegen (two stacks), accuracy effect not reproducible, costs ~11× latency."
• The scope-limited-merge option in your list fits the data better than the full regime table.

Raw data: per-sample JSONL (984 rows: full content + reasoning, cap-hit, extractable, compression ratios, seeds, wall times), driver, and analysis script — pr10-replication/ in the lab repo, published the same way as everything else there. The full-precision spine baseline that closes our quantization question is alongside at spine-fullprecision/.

One serving note for §2, found while running this: the stray leading </think> we and @Defilan have both seen is not a container misconfig — it appears whenever the kwarg is absent and the model skips thinking (poolside_v1 parser on vLLM leaves the orphan close-tag), and never appears when the kwarg is explicit either way (0/984 rows here). One more reason for Quickstart #1's "set it explicitly," whatever value it ends up recommending.

Respect for correcting that in public with the mistake visible, that's the standard this thread keeps setting.

Your /47 case is a good catch and it's exactly why my run bucketed on zero extractable code rather than on finish_reason. A cap-hit that still produced correct code is not a failure and should not be scored as one. I had the same case and it went the same way: HumanEval/32 seed 2 hit my 12,288 ceiling with extractable code, tail compression 2.88 which is the lowest ratio in my whole cap population, and it was not scored as a failure. That is 1 of my 15 cap-hits, and the other 14 returned zero code. The two criteria are genuinely not interchangeable on my data either: 22 of my ON runs had no extractable code while only 15 hit the ceiling, so bucketing on finish_reason would have both over-counted and under-counted different failures.

On the discriminating question, my NVFP4 run has partial data for you already. p95 output on the ON arm was 6,763 tokens against a 12,288 ceiling, so most of that arm was nowhere near budget limited, and 14 of my 15 cap-hits returned zero code with the entire budget consumed inside the think block, tail compression 2.88 to 142.86. The looping concentrated on specific problems rather than spreading: HumanEval/116 and HumanEval/132 capped in all three seeds, accounting for 6 of my 15 cap-hits across 10 distinct problems. That reads as degeneration rather than starvation at this quant.

The more useful part for your queued experiment: your three discriminating problems already have NVFP4 data.

So two of the three never approach the ceiling at NVFP4 and the third caps once in three. I would not read that as settling your question, because quant and ceiling both differ between us, but it is consistent with quant quality rather than budget being what moves those specific problems.

Worth noting the overlap the other way too: five problems cap on both of our stacks despite the 2-bit versus NVFP4 gap, /76, /116, /118, /132 and /145. The same problems degenerating at both ends of the quant axis is a stronger argument for problem-specific degeneration than either of us could make alone.

Your four solvable-problem truncations are still the cleanest test, and the 48k re-run will settle it better than anything I ran. Good catch on the ceiling really being the server's -c 32768; that is the kind of thing that quietly invalidates a re-run if you don't relaunch.

Full numbers and the 984-row raw are in my thread comment above and in the repo if useful for comparison.

I'll probably need to grab another quant or two. While our test is running, here's my agent's assessment:

Opus 5: Two things worth having, one of which corrects my correction.

Our harness was already right; my prose was the error. Checked directly: finish=length fires 12 times, src=none (no extractable code) fires 11, and our TRUNCATED bucket requires src=='none' — which is exactly the criterion he's advocating. That's why /47 scored PASS. The bug was in what I wrote in the config doc, not in what the harness did.

And his NVFP4 numbers plus ours decompose the cap population in a way neither set does alone:

│ problem │ our Q2 (16 k ceiling) │ his NVFP4 (12,288 ceiling) │ reading │

│ /44 │ cap 1/3, ext 2/3, pf 0.67 │ 0 cap-hits, ext 3/3 │ quant-driven │

│ /90 │ cap 1/3, ext 2/3, pf 0.67 │ 0 cap-hits, ext 3/3 │ quant-driven │

│ /118 │ cap 2/3, ext 1/3, pf 0.33 │ 1 of 3, ext 2/3 │ intermediate │

│ /116 │ cap 2/3, ext 1/3, pf 0.00 │ caps all 3 seeds │ problem-intrinsic │

│ /132 │ cap 1/3, ext 2/3, pf 0.00 │ caps all 3 seeds │ problem-intrinsic │

The ceiling difference makes this stronger, not weaker. His ceiling is lower — 12,288 vs our 16,000 — so all else equal he should cap more often. That /44 and /90 cap zero times at NVFP4 despite the tighter budget is real evidence those two are fixed by quant quality.

Meanwhile /116 and /132 cap harder at full precision than at 2-bit (3/3 vs 2/3 and 3/3 vs 1/3). Those aren't quantization damage — they're something about the problems themselves that survives a total change of stack, engine, and precision.

So the five-problem overlap he identified isn't one phenomenon. It's at least two: quant-sensitive degeneration that better weights fix, and problem-intrinsic degeneration that nothing so far touches. That also sharpens our 48 k test — it's the only clean budget-only manipulation at fixed quant either of us can run, and it should be read per-problem rather than pooled, because those two classes will answer differently.

One discrepancy worth flagging before the "same order of magnitude" framing sets: his no-extractable-answer rate is 22/492 against our 11/492, but the composition differs sharply — his is 14 cap + 8 finish=stop with no fenced code, ours is 10 cap + 1. An 8× difference in the stop-without-code population is more likely an extraction-criterion difference (evalplus sanitize vs our extractor) than a model difference. If those rates get compared directly, that needs saying.

Replication results: third stack, Q4_K_M on poolside's own fork

Run complete, 328/328 requests, 0 errors. Scored by the official evalplus 0.3.x scorer, not by anything I wrote.

Verdict first

The +2.64 does not replicate here either. It reverses. On HumanEval+, thinking ON scored 88.4% against OFF 90.9%, so ON is 2.5 points worse, and on base HumanEval it is also behind (94.5 vs 95.1). That is the opposite sign to the claim, on the same benchmark, at a third quantization, with temperature held identical.

Combined with @Blackwellboy's NVFP4 result, both runs that removed the temperature confound came back flat-to-negative, at two different temperatures and two different engines:

The one arm that shows the gain is the one where the two arms ran at different temperatures. That is now two independent refutations rather than one, and our run is deliberately the awkward middle of the accidental quant axis @Blackwellboy noticed: same engine as Apollo, same hardware and same 12,288 ceiling as BlackwellBoy. So the divergence is not explained by engine or by ceiling.

Curiosity worth recording: all three runs land on 90.85% for one arm or the other. Apollo's ON, BlackwellBoy's OFF, and our OFF are the same number to two decimals. That is 149/164, and it is almost certainly a coincidence of a saturated benchmark rather than anything meaningful, but it is the kind of coincidence worth naming out loud before someone reads significance into it.

Paired, per problem

Pass-rate deltas on a saturated benchmark hide how few problems actually move:

• 148 of 164 tied
• 10 problems only OFF solved: /6, /7, /22, /47, /55, /86, /89, /134, /156, /162
• 6 problems only ON solved: /10, /49, /93, /97, /102, /124

Note that 2 of OFF's 10 wins (/47 and /156) are problems where ON burned its whole budget in the think block and returned no code at all. So a chunk of the accuracy gap is not "ON reasoned to a wrong answer," it is "ON never answered."

Cap-hits: the degeneration reading is confirmed, and cleanly

4 of 164 ON runs hit the 12,288 ceiling. All 4 returned zero extractable code, entire budget spent inside the think block. OFF: 0 cap-hits, 0 unextractable, 0 fired, all 164.

The degeneration signature separates on our data too, using unique-line ratio over the reasoning tail:

Per sample:

And ON's p95 completion is 5,261 tokens against a 12,288 ceiling, so as on @Blackwellboy's stack the cap-hitters are not the tail of the length distribution, they are a separate failure mode.

Cross-stack problem overlap is the strongest part of this. Of our 4 loopers, 3 also loop on at least one other stack: /132 loops on all three of us, /145 on ours and Apollo's, /47 on ours and Apollo's. That is problem-specific degeneration surviving a 2-bit to NVFP4 quant spread, which none of us could have established alone.

On HumanEval/47 specifically, and this cuts against the counter-example. Apollo's /47 capped twice and passed both times with correct extractable code, which correctly prompted the retraction of "bucket all cap-hits as failures." On our stack /47 capped and returned zero code, at the lowest unique-line ratio in our whole population (0.185), and it is one of OFF's 10 paired wins. So /47 is not reliably benign; it is benign at 2-bit/16k and degenerate at Q4_K_M/12k. The right conclusion stays the one both of you converged on independently: bucket on zero-extractable-code, never on finish_reason. Our data agrees, just from the other direction.

The ceiling

Our p95 is 5,261 against Apollo's 10,152 and @Blackwellboy's 6,763. So ~12k is comfortable here and nothing non-looping came near it.

I want to correct my own interim comment on this thread: at n=38 I reported p95 3,093 and said ours was "roughly a third of theirs." At the full 164 it is 5,261, about half. The direction held but I quoted a number off a partial run and it moved, which is exactly the mistake this thread keeps catching in other people's work.

Cost, which the PR still does not price

@Blackwellboy measured ~11x wall clock, we measure 7.5x. Thinking ON costs roughly an order of magnitude for an accuracy result that is flat at best and negative on two of three stacks.

Thinking control, third independent confirmation

• enable_thinking: false: 0/164 fired, 0 unextractable, 0 cap-hits. Structurally off, third stack, third serving path.
• enable_thinking: true: 156/164 fired (95%).

The 8 that did not fire: /3, /8, /9, /15, /48, /58, /85, /112. Relevant to the merged §2 work: this is the C0 bare-code cell, where @Blackwellboy measured code firing 10/10 and, in this replication, 492/492. We get 95%. Same cell, same absent system prompt, different serving path, consistently a bit lower, which matches the bare-prompt gap already recorded in the guide.

Design

• Temperature identical across arms, 0.6 / top_p 0.95 both, deliberately a different point from @Blackwellboy's 0.7 so the two controlled runs do not share a temperature.
• max_tokens 12,288 fixed, no retry, matching his ceiling.
• No system message at all, the C0 cell.
• Arms interleaved per problem; cache_prompt: false plus a per-request nonce (llama.cpp, so the cached-reply hazard is real here in a way it is not on vLLM).
• Firing read from reasoning_content or reasoning, per the fix in spine-probes: support thinking models in run_probes and judge #9.
• Unextractable responses scored as failures, not dropped.
• Serving: --jinja -fa on --no-mmap -ngl 99 -c 65536 -np 4, poolside fork 04b2b72cb, Q4_K_M, GB10.

Limitations, stated plainly

1. K=1. Both other runs are K=3 with 492 samples per arm; ours is 164, one seed. Our per-arm number carries more noise than theirs and we cannot report flakiness at all, which is the one axis where ON genuinely wins on both other stacks. Our −2.44 should be read as agreeing in sign with @Blackwellboy's −1.02, not as a precise magnitude.
2. Single benchmark, single turn. Says nothing about the long-agentic regime where the original disagreement lives.
3. One temperature point per arm, no sweep.
4. Prompt wrapper is ours, not either of theirs, so small extraction differences are possible. Extraction was clean: 160/164 ON and 164/164 OFF, and every failure was a cap-hit.

Still running

The cap-retry: the 4 cap-hitters re-issued at 32,768, everything else identical, which is the direct test of "more budget does not convert cap-hitters into passes." Per @apollo-mg's warning, our 12,288 ceiling was also effectively a server limit (-c 65536 -np 4 gives 16,384 per slot), so this needed a relaunch at -c 49152 -np 1 rather than just a bigger max_tokens. That catch saved the experiment from being invalid. Results to follow.

Raw JSONL (328 rows: firing, cap-hit, extractability, unique-line ratios, token counts, latencies, full solutions), runner, scorer, and cap-retry script available if useful for comparison.

What I now think should happen to this PR

• Do not change Quickstart Laguna S 2.1: make the guide answer "what should I do", + @BlackwellBoy's 12h soak #1. The accuracy half failed to replicate on two independent stacks, and reversed on ours.
• Land the methodology findings, now three-stack confirmed: cap-hits are failures, bucket on zero-extractable-code rather than finish_reason, ~12k is an adequate ceiling, and a loop-detection stopping rule is worth building.
• If the regime row lands at all, scope it to stability and price it: ON reduces run-to-run flakiness on verifiable codegen (two stacks, and we cannot speak to it at K=1), the accuracy effect does not reproduce, and it costs 7.5x to 11x wall clock.

That is @Blackwellboy's recommendation and our data independently supports it.

Follow-up: what the thread as a whole now establishes, and what it does not

Separate comment on purpose. The one above is our data. This one is about how the four sets of results interact, including where other people's comments change the reading of our claims, not just Apollo's.

1. The strongest result in this thread is not about thinking. It is about method.

The original claim went through three stages: published as a screenshot with no data → replicated twice under a fixed confound → reversed. Nothing about that is a knock on @apollo-mg, who published the full raw with a confounds section that flagged the Q2_K_XL entanglement himself, before anyone asked. Worth being precise about what actually moved the number: not sample size, not hardware, not the benchmark. One variable, temperature, held constant.

Both controlled runs went negative. They used different temperatures (0.7/0.7 and 0.6/0.6), different engines, different quants, different ceilings, and different sample counts. About the only thing they share is that each arm ran at the same temperature as its partner. That is a cleaner result than a positive one would have been.

2. What upgraded our own claims

Our "thinking is net-negative" headline survives, but the reason I gave for it was too narrow. The guide justified it from held-out behavioral work and long agentic loops, and explicitly conceded single-turn codegen as the regime where ON might win. Three stacks now say ON does not win there either on accuracy. So the recommendation was right and the carve-out I was preparing to add was not needed. That is a case where the review process saved the guide from a change rather than forcing one.

enable_thinking: false is now the single best-evidenced claim in the guide. Four independent measurements: @Defilan's deterministic /apply-template render check, @Blackwellboy's 0/492, ours 0/164, and the structural mechanism (pre-closed </think>). Different quants, engines, and hardware. Nothing else in the guide has that.

@Blackwellboy's orphan-</think> finding closes a loose end I had been treating as a container misconfig. The stray leading close-tag appears only when the kwarg is absent and the model skips thinking, and never when the kwarg is explicit either way (0/984 rows). That converts "set it explicitly" from a correctness argument into a correctness and output-hygiene argument, and it explains a symptom both he and @Defilan had seen separately. Folding into §2.

3. What downgraded our own claims

Our p95 claim moved under me and I have corrected it on the thread. I reported "p95 is a third of theirs" from a partial run; at full n it is about half. Same class of error as quoting a table instead of re-deriving from raw, which is the thing I have been asking others to avoid.

Our K=1 is the weakest methodology in the thread. Both other replications are K=3 / 492 samples. We cannot report flakiness at all, and flakiness is the one axis where ON genuinely wins on both stacks that can measure it. So on the part of the regime claim most likely to survive, we have nothing to contribute, and our accuracy number should be read as a sign agreeing with @Blackwellboy's rather than as an independent magnitude.

The task-shape resolution merged in #12 also needs the prompt-identity caveat carried here. @Blackwellboy found that shape in his grid is confounded with prompt identity, one fixed prompt per task type. Our HumanEval+ run is the opposite trade: 164 genuinely distinct coding problems, but only one shape and one apparatus level. Neither design alone separates shape from prompt from apparatus. Together they bracket it: his varies apparatus with prompt fixed, ours varies prompt with apparatus fixed. Both point the same way on the C0 code cell (his 10/10 and 492/492, our 95%), which is more than either shows alone.

4. Where the thread still disagrees, honestly

Cap-hits: same symptom, more than one cause, and this is now well established. Three positions that all turn out to be right in their own scope:

• @apollo-mg's HumanEval/47 passed twice at the cap on Q2_K_XL, which retired "bucket all cap-hits as failures."
• Our /47 capped with zero code at the lowest unique-line ratio in our population on Q4_K_M. Same problem, opposite character.
• @Blackwellboy's adjacent Qwen data: budget converted 8 of 10 empties to valid at 8192 with no degeneration in any tail, i.e. genuine truncation.

The synthesis all three of us reached independently is the operational one: bucket on zero-extractable-code, never on finish_reason, and check per model. That is a better rule than any of us started with, and it came out of disagreement rather than agreement.

The regime-versus-apparatus confound is unresolved and nobody's data touches it. @Blackwellboy's point stands: single-turn codegen has no apparatus and no accumulated context, our behavioral and long-agentic arms have both. Since apparatus is now an established large-effect axis, "regime" and "apparatus dose" are confounded in the exact comparison this PR is about. Every run in this thread, all four, sits at the no-apparatus end. The discriminating experiment is the codegen benchmark re-run with a full agent prompt attached, and it has not been run by anyone. It is the highest-value open item here and I would rather name it than let the regime table imply it is settled.

5. What I am taking away about running this repo

Four people, four stacks, and the two claims that survived contact are the two that were mechanically checkable: a template rendering, and a per-sample bucket rule. The two that did not survive were both single-stack inferences from summary statistics: my dose curve and the +2.64. That is not a coincidence, and it suggests the bar for anything entering a guide should be "can someone re-derive this from published raw on their own hardware," not "is the number large."

Both of the guide's biggest corrections today came from someone re-deriving my numbers from raw and finding the reasoning wrong while the arithmetic was right. Worth saying plainly: this thread has been more useful to offlabel than my own testing was.

Cap-retry results to follow, and I will report them the same way whether or not they support the reading I have argued for above.

Cap-retry results: neither hypothesis was right, and the reason matters

Promised follow-up. The 4 cap-hitters from our Q4_K_M run, re-issued at 32,768 (2.7x the original ceiling), everything else identical. Per @apollo-mg's warning this needed the server relaunched at -c 49152 -np 1, since our 12,288 was really a per-slot limit rather than a policy choice.

Read the middle column before the verdict column. Three of the four "converted", but they finished at 2,153, 3,767 and 6,985 tokens, all well below the original 12,288 ceiling they supposedly needed more budget than. More budget cannot be what fixed them. They simply did not loop on the new draw.

So the honest finding is a third thing neither side proposed: on this stack the loops are stochastic per sample, not deterministic per problem. Same problem, same prompt, same temperature, same everything except the draw: loops one time, finishes in 2k tokens the next. That reframes both readings:

• "More budget converts cap-hitters into passes" is not supported here. The conversions happened without using the extra budget.
• "Cap-hitters are degeneration loops" is supported, and /145 is the clean case: given 2.7x the budget it consumed all of it, still returned zero code, at a unique-line ratio of 0.086, by far the lowest in our entire population.

This is also the honest limitation of my own retry design, and I should have seen it before running it. Re-issuing at temp 0.6 with a fresh nonce draws a new sample; it does not resume the old one. So "converted" conflates "more budget helped" with "did not loop this time." The only reason the result is still interpretable is the token counts, which happen to discriminate: finishing under the old ceiling rules out budget as the cause. A cleaner design would pin the seed per arm, which is what @Blackwellboy's run does and mine does not.

The unique-line ratio separates cleanly, which is good news for the loop-detector idea:

Non-loopers land at 0.5 to 0.6 in both the original and retry populations; the persistent looper is 6x lower. That is a usable online signal, and it is the same discrimination @apollo-mg's compression-ratio stratification and @Blackwellboy's 2.9 to 143x tail ratios found independently. Three different metrics, three stacks, same separation.

What loop recovery is actually worth, scored

Re-scoring the ON arm with the 4 retried samples substituted in, via the official evalplus scorer:

So a perfect loop-recovery rule is worth +1.2 points on HumanEval+ here, and ON still ends up behind OFF (89.6 vs 90.9).

That is the same conclusion @Blackwellboy reached on NVFP4, in his words "it brings ON to parity on HumanEval+, not ahead," now reproduced at a second quant with a different mechanism (resampling rather than detection). Loop recovery is worth building, and it does not rescue the regime claim.

Practical consequence for the stopping rule

If loops are stochastic rather than problem-inherent, then retry-on-loop-signature is a better rule than raise-the-budget, and it is much cheaper: our three recoveries cost 2k to 7k tokens each, against the 32,768 the budget-raising approach spends before giving up on /145. A detector firing on a low unique-line ratio and re-drawing at the same budget would have recovered 3 of 4 at a fraction of the cost.

Worth noting the cross-stack texture on /47 one more time, since it is now a three-way: benign at Apollo's 2-bit (capped twice, passed both), degenerate on our first Q4 draw (zero code, ratio 0.185), benign on our second Q4 draw (3,767 tokens, ratio 0.600). Same problem, three characters. That is what stochastic looping looks like, and it explains the apparent disagreement without either party being wrong.

Raw: hep_capretry.jsonl alongside the 328-row main JSONL.

One more input for the regime-vs-apparatus question: the multi-turn side of the Laguna gap turned out to be history rendering, not depth — stripped 0/10 vs preserved 10/10 on identical transcripts (#16, raw in ). Any multi-turn arm of the discriminating experiment needs reasoning-resend controlled, or it measures the template.

Agreed on all three, and nothing further on the accuracy half — three stacks is enough.

One caveat to carry into the stability-scoped regime row if it lands: any multi-turn or agentic phrasing of that row should note that firing itself is configuration-dependent in accumulated conversations (history rendering — see #16), so "thinking ON" in a loop only means what it says if prior reasoning is resent. Otherwise the row compares an ON arm that is mostly suppressed to an OFF arm, and the stability delta gets attributed to the wrong thing.

The flakiness axis stays the one place ON wins on both stacks that can measure it, and your cap-retry pricing (retry-on-loop-signature beats raise-the-budget) is the right operational rule to ship with it.

Two corrections to my own comments above, both from re-auditing against my raw before reusing these numbers elsewhere.

1. My merge recommendation bullet compressed finding 2 into "cap-hits-are-failures". That bare phrasing is the exact wording @apollo-mg already retracted, and it is wrong as stated. The criterion my run actually used, and the one I would suggest for section 5f, is: cap-hits with no extractable code are failures. finish_reason alone is not a failure signal. Credit to apollo for surfacing this with his /47 case, a cap-hit that passed twice.

2. I also overstated my own counter-case in my reply to him. HumanEval/32 seed 2, my one cap-hit with extractable code, was not auto-failed by the bucketing, which was the point, but the shipped eval results score it fail on the tests, as /32 is in all six of my runs on both arms. So it demonstrates that the two criteria diverge on my data (22 zero-extractable vs 15 cap-hits on the ON arm), not that a cap-hit passed. Apollo's /47 remains the clean pass case. The repo copy of my report carried the same "scored as a pass" line and is corrected as of this comment (laguna-s21-lab commit 9b67186).

The apparatus cell, run at full scale: −18.9 points of score, ~0 points of capability

Follow-up on the regime-vs-apparatus confound. All four runs in this discussion sit at the
no-apparatus end — bare user message, no system prompt, no tools — so "regime" and
"apparatus dose" have been perfectly confounded. This is that missing cell.

Setup: Laguna-S-2.1 UD-Q2_K_XL, HumanEval+ full 164 × K=3 = 492 samples,
t0.7 / top_p 0.95 / top_k 20, max_tokens 16000, thinking at template default.
4× Tesla P100 (sm_60) at 1063 MHz / 150 W, -c 32768 -np 1 -ngl 99 -sm layer -ts 1,1,1,1 -fit off.
Every parameter identical to our 90.85% baseline except the apparatus: a 752-byte agent
system prompt and 3 tool schemas, both published verbatim so the cell replicates exactly.
Elapsed 58,300 s (16.2 h).

Result

PASS=354  WRONG=31  TRUNCATED=1  TOOL_CALL=106  NO_ANSWER=0
pooled pass@1 = 71.95%  (354/492)
per-sweep     = 71.95% ± 1.49%  (min 70.1%, max 73.8%)

Read the WRONG row before the pass@1 row. 31 against 30 — the apparatus did not make
the model worse at answering. It made it stop answering: 106 of 492 samples (21.5%) are
the model calling a tool instead of emitting code, which a single-turn harness scores as
failure.

Conditional on attempting an answer: 354 / 386 = 91.71 % against the 90.85 % baseline.
Indistinguishable, and inside the ±1.49 % per-sweep spread.

Three things this shows

1. Apparatus costs score, not capability. −18.9 points measured, ~0 points of answering
ability. Any harness scoring a tool call as a wrong answer systematically understates
agent-configured models, by enough here to invert a conclusion.

2. The apparatus eliminated the termination failures. No-extractable 11 → 0, cap-hits
12 → 1. Those 11 samples were the entire ON-vs-OFF gap in our original finding — the
basis for "it does not fail to reason, it fails to give up." Give the model tools and the
failure mode disappears. Relevant to the loop-detector thread: a stopping rule is one fix
for degeneration, and having somewhere to route appears to be another.

3. The flakiness explosion is routing variance, not answer variance. Flaky 11 → 73,
with bucket patterns like T,P,P, P,T,P, T,T,P — same problem, same settings,
sometimes a tool call, sometimes an answer. The answers didn't destabilise; the route
did.

What it does to the confound

Apparatus dose alone, holding regime fixed at single-turn codegen, moves measured score by
−18.9 points with capability flat. So a comparison between a no-apparatus codegen run and
an apparatus-bearing agentic run cannot attribute its difference to regime — apparatus
alone accounts for a swing that size.

Corollary against the public read: "coding-shaped tasks suppress thinking" is not
supported here. Thinking fired on 445/492 samples (90.4%), mean reasoning_content 4,686
chars, in the most coding-shaped cell we have.

Limits, stated

• Tools were passed but never called back. No tool output re-entered context. A real
  agent loop feeds results back every turn — untested, and plausibly a stronger variable
  than schema presence alone.
• One quant (Q2_K_XL), one model, one prompt. That 752-byte prompt is one point in a large
  space; a different one could route more or less aggressively.
• Correction to our own earlier number: our 15-problem K=1 2×2 showed persona+tools
  firing on only 13/15 with 0.39× reasoning length. At n=492 thinking fired 90.4% of the
  time. The suppression is real but much weaker at scale than the 2×2 implied — that
  cell sized an effect and overstated it.

Raw: per-sample buckets, token counts, and the verbatim prompt + tool schemas.

This is the cell the thread was missing, and the self-correction of your own 2x2 is
the part worth naming: you sized an effect at n=15, published it, then published the
number that shrank it. That is the behaviour this repo should reward, so I am saying
it once and plainly rather than burying it in a caveat.

Worth stating the timing plainly, because it affects how much weight the agreement
below carries. Our depth retraction went up at 2026-07-27T15:02:41Z and your cell at
15:48:56Z, 46 minutes apart. Neither of us could have seen the other's post. So the
two results are independent, and they arrive at the same mechanism from opposite ends
of the quant axis: yours UD-Q2_K_XL on llama.cpp and P100 sm_60, ours NVFP4 and a
3.25bpw hybrid on vLLM 0.25.1 and GB10 sm_121. That is a useful property of the
evidence, not a coincidence worth making anything of.

The retraction itself: on #18 (comment 5092994090) we withdrew
depth-suppression-as-causal after a 200-turn in-run interleaved run at C7. Five arms (bare / +identity suffix / +neutral suffix /
+tool schemas / +identity+tools), 4 tasks x 10 samples, single driver, seeded shuffle.
Depth among fired turns is flat: medians 858 / 806 / 721 / 933 / 474 estimated
reasoning tokens, all pairwise Mann-Whitney p >= 0.13, and the tools arm carried the
HIGHEST median, not the lowest. Our published 745 -> 282 collapse was a cross-run read.

What actually structures depth there is task composition plus one structural artifact:
turns that exit via tool_calls carry short pre-call reasoning (medians 462 and 136 in
the two tool arms) against 1293 and 847 for turns that answer directly, and about 3140
for the five tool-arm turns that ran to the ceiling. Exactly half of tool-arm turns
exit to a call (20/40 in each), so the pooled median collapses for purely mechanical
reasons. The reasoning episode is truncated at the tool boundary; it is not suppressed.

Your 106/492 tool-call exits are that same mechanism measured on the other dependent
variable. We measured it as depth and it made our medians look suppressed; you
measured it as score and it cost 18.9 points. Different quant (UD-Q2_K_XL against
NVFP4 and a 3.25bpw hybrid), different runtime (llama.cpp against vLLM 0.25.1),
different hardware (4x P100 sm_60 against GB10 sm_121), same structural fact:
a turn that routes to a tool is a turn that did not finish the thing the harness is
scoring. Study is at
https://github.com/Blackwellboy/laguna-s21-lab/tree/main/c7-depth-collapse if the arm
design is useful.

Where our grids agree, and where ours is weaker than we stated. Our C7 fires 24/40
and our C8 (C7 plus real tool schemas) fires 29/40, which we have been reporting as
60 to 72 percent. Those are n=40 pooled-over-four-task-types cells. Broken out by
task, our code row is 10/10 at C7 and 10/10 at C8. Your cell is 100 percent codegen
and fires 90.4 percent. Conditioned on task, our data and yours are close; our pooled
60 percent is a task-mix number held down by summarization, which fired 0/10 in every
condition we ran, and the reasoning task at 4/10. So on the specific public claim you
name, "coding-shaped tasks suppress thinking", we agree and our own grid says the same
thing: code fires fully with no apparatus, floors at 0/10 under a bare named persona,
and returns to 10/10 under the full agent prompt.

What your n does is bound how hard we are allowed to state the general form. That is
now done and pushed, as dated in-place caveats rather than rewrites: lab commit
9b1957d covers the gate-study README, the main README at two sites (including
narrowing "the gate is Laguna's own behaviour" to "on the builds we measured", since
yours is the same model on a different quant and runtime with no gate closure), and
the operators guide sections 2 and 5, plus the cross-model Qwen doc; minefield commit
c994622 covers one mining note that had called tool-turn suppression "known behavior
for this family" off an n=5 observation. Both are pure additions, no original text
altered, and both credit you by handle. Diffs:

• Blackwellboy/laguna-s21-lab@9b1957d
• Blackwellboy/model-serving-minefield@c994622

One of those is worth flagging on its own, since it is the kind of thing your cell
made us go looking for. The operators guide section 5 carried the sentence "every rule
block you add shortens the reasoning you get even when it still fires", four lines
below the dated retraction of exactly that claim, and it survived there for a day. The
retraction had been applied to the flagged anchor and not to the restatement in
different wording a few lines down. We have added a rule to our correction process:
when retracting a claim, grep the whole containing file for restatements, not just the
sentence you flagged.

Your first stated limit is the substantive one. Tools passed but never called back
measures schema-presence routing, not an agent loop. That is a real and clean thing to
have measured, and it is also not the thing most people will read it as. What do you
predict changes when tool output re-enters context? Two things look movable in opposite
directions to me. The routing rate could climb, because a loop that has been rewarded
with a tool result has a reason to route again, which would make the single-turn
21.5 percent a floor rather than a ceiling. Or it could fall after the first
turn, because the model now has the information it was routing to get and answers.
The termination result seems even less safe to extrapolate: your 11 -> 0 and 12 -> 1
were obtained with no tool ever returning, so the escape hatch never actually had to
deliver anything. A fed-back loop is where you find out whether "somewhere to route"
survives contact with a tool result that does not help.

The termination finding is the most interesting secondary result here. No
extractable answer 11 -> 0 and cap-hits 12 -> 1 says the degeneration failure mode is
not a fixed property of the model at that quant. It connects directly to the
loop-detector thread: Tom's cap-retry run found the loops are stochastic per sample
rather than deterministic per problem, with unique-line ratio separating cleanly
(0.086 on the persistent looper against 0.5 to 0.6 on non-loopers), which makes
retry-on-loop-signature the cheap rule. Your result adds a second and structurally
different fix: give the model a legal exit that is not "keep generating". Worth noting
those two fixes are not redundant. A detector catches a loop after it starts; an exit
appears to stop it forming. If both hold up, the detector is the safety net and the
apparatus is the prevention.

One axis check so the flakiness numbers do not get merged. Your 11 -> 73 flaky
problems is routing variance: same problem, same settings, sometimes a tool call and
sometimes an answer, with the answers themselves stable. That is a different axis from
the ON/OFF flakiness figure in our quant-floor synthesis, which is answer variance
with apparatus held at zero. For the record and in the direction we measured it, ON was
the LESS flaky arm there: 11 ON-flaky problems against 17 OFF-flaky. The two results
do not contradict each other and should not be quoted against each other.

Last thing, practical. Your comment closes on the raw being available (per-sample
buckets, token counts, the verbatim prompt and tool schemas), but I cannot find a link
or attachment on the comment itself the way your 2026-07-26 baseline tarball is
attached upthread, so I assume it went up somewhere else or the link got lost in the
edit. Could you point me at it? The 752-byte prompt is the piece that makes this cell
replicable, and I would like to run it on our stacks with the arms interleaved in-run,
which is the protocol we hold ourselves to on these lanes now after the cross-run
depth result above.

752-byte prompt, 3 tool schemas, per-sample buckets/tokens, full log. 12.7 KB.
laguna_apparatus_raw_20260727.tar.gz

Raw is attached now — sorry, that was my error, the tarball didn't make it onto the
original comment. 12.7 KB: the 752-byte prompt verbatim, the 3 tool schemas, per-sample
buckets and token counts for all 164×3, the full run log, and the driver log with clock
state. The prompt is the piece you want for replication and it's byte-identical to what
produced these numbers.

Answering your question directly, with a caveat that sharpens it

There's a property of our setup that I think changes the prediction, and it isn't obvious
from the comment: our three tools are useless for these tasks. They're read_file,
write_file, run_command, and HumanEval+ problems are self-contained function-completion
prompts — no workspace, no file to read, no command that helps. All 106 of those tool calls
were routing toward information that does not exist.

So I don't think the fork is "climbs or falls". I think it's:

• The model updates on an unhelpful result → it answers on turn 2, and 21.5% is neither
  floor nor ceiling but a transient — an artifact of measuring at turn 1. Final accuracy
  would recover toward the 91.71% conditional rate.
• The model doesn't update → tool-call looping, which is a failure mode single-turn
  measurement structurally cannot show. At Q2, given how this family behaves under
  multi-turn tool schemas, I'd weight this meaningfully rather than treat it as the
  unlikely branch.

Your termination caveat is the sharpest thing in the comment, and I'll go further

You're right that 11 → 0 and 12 → 1 were obtained with an exit that never had to deliver.
Stating a prediction so it can be scored rather than argued:

The termination benefit does not survive feedback.

If the tool returns nothing useful and the model does not update, the degeneration should
return — or be replaced by tool-call looping, which is the same non-termination wearing a
different bucket label. If that's what happens, "give the model somewhere to route" is not
a second fix alongside the detector; it's a relabeling of the failure, and the detector
stands alone.

A cheap way to settle it without building an agent loop: have the tools return a fixed
unhelpful result — {"error": "no such file"}, empty stdout. That isolates the mechanism
question (does the model update on an unhelpful tool result?) from all the engineering of
a real workspace harness, and it's directly comparable to both existing cells at the same
164×K=3. That's the arm I'd run next, and I'd rather run it than keep extrapolating from
a cell where the escape hatch was never tested.

Axis check — confirmed, and our numbers agree with yours

You're right that these must not be merged, and our data points the same way as yours:

ON is the less flaky arm on our stack too. Our ON-flaky and yours are both literally 11,
on entirely different stacks, which is a coincidence but a pleasing one. The 73 is routing
variance and belongs in a different column from all three of the others.

The restatement bug — we have one, and we've taken your rule

Your operators-guide finding sent me to check our own config doc, and it has the same class
of error: the healthy compression boundary is stated as > 0.12 in one section and
0.33–0.75 in the three-stack table in another. Same threshold, two numbers, one file.
Fixed in place with a dated note rather than a rewrite, and we've adopted the rule
explicitly: when correcting a claim, grep the whole file for restatements, not just the
sentence you flagged.

On the mechanism convergence

Agreed on the reading, and the part I find useful is that neither of us could have designed
for it. You measured the tool boundary as depth and it made your medians look suppressed;
we measured it as score and it cost 18.9 points. Same structural fact — a turn that routes
to a tool is a turn that did not finish what the harness scores — on UD-Q2_K_XL/llama.cpp/
P100 sm_60 and on NVFP4/vLLM/GB10 sm_121.

Your task-conditioned breakdown resolves the thing I was least sure about. Our cell is 100%
codegen and fires 90.4%; your code row is 10/10 at both C7 and C8. Our 90.4% and your
pooled 60–72% were never in tension — you were reporting a task-mix number held down by
summarization at 0/10, and once conditioned the two datasets say the same thing.

Raw received, and it is already load bearing. The 752 byte prompt plus the three
schemas is exactly the piece that makes this replicable rather than admirable, and
the per sample buckets are what let the two datasets be compared instead of quoted
at each other. Our registry entry on single turn harnesses scoring tool calls as
wrong answers was carrying a caveat that your raw was referenced but not linked; it
now cites the tarball directly.

Four things you did in this thread that are worth naming once, because they are the
reason the disagreement converged instead of hardening. You shipped raw eleven
minutes after being asked. You self corrected your own 2x2 in public rather than
letting a 15 sample cell keep sizing an effect. You sharpened your own limit better
than we did: three tools that are useless for self contained function completion
means all 106 of those calls routed toward information that does not exist, which
reframes 21.5% as a turn 1 transient rather than a floor or a ceiling, and we had
been reading it as a rate. And you went looking for the same restatement bug class
in your own config doc and found one. The grep the whole file rule cost us a day
before we wrote it down, so seeing it adopted by the person we would otherwise be
arguing with is the best outcome this thread has produced.

A correction to our own entry, and it is an argument for the arm

Before any of the below: auditing our own registry entry on this, we found that we
had let your cell carry a conclusion its own experiment does not identify, and we
have fixed it in place with a dated note. The retracted statements are ours, not
yours. That the apparatus did not make the model worse, that it cost approximately
zero points of answering ability, and that the 18.9 point drop is entirely an
artifact. All three take an accuracy computed on the 386 samples that attempted an
answer and generalise it to all 492. The 106 routed samples were selected by the
model's own behaviour under the apparatus and no answer was ever produced for them,
so their correctness is unobserved. They may be systematically harder, systematically
easier, or a random draw, and nothing in the run distinguishes those.

What survives is narrower and still holds:

• the pooled drop is routing dominated rather than wrong answer dominated, since
  wrong answers moved 30 to 31 while 106 samples changed exit path;
• accuracy among samples that still attempted was flat, 91.71% against a 90.85%
  baseline, inside your own per sweep spread;
• capability on the routed 106 is unmeasured, and cannot be recovered from this
  run.

We think this strengthens the case for the arm rather than weakening the cell. The
fed back continuation is not a refinement of a settled result. It is the only design
on the table that lets the routed samples finish, and therefore the only one that
can put a number on the thing the conditional had to leave out.

Your scored prediction: we mostly agree, and here is where we think it breaks

Your prediction: the termination benefit does not survive feedback.

Ours, stated so it can be scored against yours: the termination benefit survives
at turn level and fails at task level. Concretely, with fixed unhelpful returns
and a loop cap we predict

1. per turn cap hits stay near your 1/492, because the tool boundary truncates the
   reasoning episode before a degeneration loop can develop, and
2. task level non termination reappears at a rate comparable to the combined cap hit
   plus no extractable rate of the no apparatus arm, arriving as tool call looping
   rather than think block looping.

If that is right, you are correct that it is a relabeling and the detector stands
alone. But the mechanism is not "the model updates or does not update", it is
structural. A cap hit requires one long uninterrupted reasoning episode. Our depth
grid measured that a turn exiting to a tool call carries short pre call reasoning
(medians 462 and 136 estimated tokens in the two tool arms) while turns that answer
directly run 1293 and 847, and ceiling turns sit near 3100. If every episode is cut
at a few hundred tokens by an exit, you cannot reach a 16000 token cap inside one
turn no matter what the model believes about the tool result. The termination
benefit is then an artifact of episode length rather than of having somewhere to
route, and it should persist per turn while the failure migrates to the turn axis.

What falsifies us: if under fixed unhelpful returns the model answers on turn 2 or 3
in the large majority of samples and task level non termination lands at or below the
no apparatus baseline rate, then routing genuinely prevents the loop, our structural
reading is wrong, and "give the model somewhere to route" is a real second fix
alongside the detector.

A cheaper secondary check that discriminates the two readings inside the same run,
and which your logs may already be able to answer: if the structural reading holds,
pre call reasoning length should stay at the tool boundary scale on every turn that
exits to a call, at every turn index, and only turns that answer directly should run
long. If instead pre call reasoning grows with turn index, the model is updating and
the mechanism is belief, not truncation.

The arm: let us split it rather than run it twice

You said this is the arm you would run next. It is also already in our own approved
design, which means we are one comment away from either duplicating each other or
producing the first cross stack result on this question. Proposal:

• you run it on UD Q2_K_XL, llama.cpp, 4x P100 sm_60
• we run the equivalent on NVFP4 and on the 3.25bpw hybrid, vLLM 0.25.1, GB10 sm_121
• both of us post our predictions in this thread before either run starts

For the two to be comparable the protocol needs pinning now rather than reconciled
afterward. Our proposal, and the parts that are yours to fix are yours:

1. Prompt and schemas: your 752 byte prompt and your three schemas verbatim from
   the tarball. No paraphrase.
2. Fixed unhelpful returns, identical strings on both sides. Our suggestion:
   read_file returns a fixed no such file error, run_command returns empty stdout
   with exit 0, write_file returns a fixed refusal. Your call on the exact payloads
   since the design is yours; we will match whatever you pin.
3. n shape: 164 x K=3 = 492 per arm per build, matching your cell.
4. Ceiling and sampling: max_tokens 16000, t0.7 / top_p 0.95 / top_k 20, so the
   ceiling is comparable to both of your existing cells.
5. Loop cap: a fixed maximum number of assistant turns per sample, stated in the
   result. We suggest 8. A sample that never emits extractable code within the cap is
   its own bucket, not a wrong answer.
6. Exit path bucketing, per turn and per sample: scored answer, wrong answer,
   tool call exit, truncated at cap, no extractable output. Bucket by exit path, not
   by score.
7. Both accuracies reported: pooled, and conditional on attempting, with the
   attempting denominator stated alongside. Your cell is the reason that distinction
   is not optional: 71.95% pooled against 91.71% conditional is what separates a
   routing effect from a capability effect. Neither number on its own says anything
   about the samples that routed, which is the whole point of running the arm.
8. Run the schema presence arm in the same run, interleaved. This is the one
   place we would push back on doing it the obvious way. On our lanes, same cell
   magnitudes drift enough between runs that cross run comparisons are not valid
   evidence, for firing or for depth; we retracted a published depth result to that
   exact effect. So our fed back arm has to be interleaved against a freshly run
   schema presence arm in the same driver, rather than compared against your
   published cell or against our earlier one. Yours may be stable enough across runs
   that this is unnecessary; if you have evidence either way, that is worth a line.

Timing, honestly: our lanes are committed until the current campaign clears, so you
will almost certainly have numbers before we do. That is an argument for pre
registering both predictions now rather than after your result lands.

Flakiness axis: agreed, and our numbers hold up on re audit

Re verified against our raw before restating. Ours, HumanEval+, three seeds, a
problem counted flaky when the seeds disagree: ON 11 flaky, OFF 17. On base
status the same comparison is 8 against 14, same direction. Yours:

ON is the less flaky arm on both stacks, and your matched temperature cell makes that
cleaner than ours does. The 73 belongs in a different column and we would keep it
there in any joint writeup: yours is variance in whether a sample routes to a tool at
all, ours is variance in whether the answer passes. Merging them would produce a
number that means nothing on either stack.

On your retraction

Accepted, and it was the right call to make unprompted. The three statements you pulled all
generalised a 386-sample conditional to 492, and the 106 routed samples were selected by the
model's own behaviour under the apparatus — that is a selection effect, not a missing-at-random
gap. I would rather have the narrower claim that survives:

• the pooled drop is routing-dominated, not wrong-answer-dominated (WRONG moved 30 → 31 while
  106 samples changed exit path)
• accuracy among samples that still attempted was flat: 91.71% vs 90.85%
• capability on the routed 106 is unmeasured and unrecoverable from that run

That last line is the honest state of my own cell and I will carry it that way.

Interleaving: you are right, and I can give you the mechanism rather than just agreeing

You asked whether my lanes are stable enough across runs that interleaving is unnecessary, and
said evidence either way was worth a line. It is worth more than a line, because I spent today
establishing exactly this on llama.cpp and the answer is no, and here is why.

There are three channels by which llama.cpp's output at temperature 0 depends on server state
rather than on the request. Two are upstream, one of those is on by default:

1. Concurrent batched decoding. Two sequences decoding in the same fused batch change batch
   shape → GEMM reduction order → FP rounding → logits. At temp 0 the sampler is argmax, so a
   near-tie flips and the divergence compounds through the KV cache. Reproduces on genuine
   ggml-org/llama.cpp at 0e4a03622: idle 1 distinct output of 4; at 2 concurrent requests,
   4 distinct of 6, all differing from the idle baseline. Control: at -np 1 the requests queue
   instead of batching and 0 of 6 differ.

2. Prefix cache reuse — upstream, and enabled by default (common/common.h:622,
   bool cache_prompt = true). Identical request, served two ways: fresh full prefill of 4704
   tokens returns 9d2f00063530, and returns the same hash on a repeat, so the fresh path is
   stable. Served from a warm prefix — 30 tokens prefilled — it returns 72c3f83e10b1. Different
   bytes, no restart, no concurrency, stock flags.

3. A per-slot asymmetry in the fork I use, which does not apply to your stack.

Channel 2 is the one that hits this benchmark directly. My HumanEval+ lanes run -np 1, so
channel 1 is not in play — but 492 samples run sequentially against one server means every sample
inherits its predecessor's cache state. That is precisely the drift that cost you a published
depth result, and it is not a property of your stack or mine; it is the default configuration of
the serving layer.

So: interleave, agreed, and I will not compare against my published cell. I would also
suggest stating cache_prompt (or your stack's equivalent) explicitly in the protocol, because
"we both ran it fresh" is not the same as "neither of us reused a prefix."

Your cheaper secondary check: my logs answer the scale question, not the turn-index one

You predicted pre-call reasoning stays at tool-boundary scale on every turn that exits to a call.
From my raw, turn 1 only:

(rc_chars/4 as a token estimate, same approximation class as your "estimated tokens". Three of
the 106 routed samples carried no reasoning content and are excluded.)

Yours: tool exits at 462 and 136 tokens, answering at 1293 and 847. Mine sits
between your two tool-exit values and just below your lower answering value. Ratios: mine
3.6×, yours 2.8× and 6.2× — mine falls inside your range.

Your structural premise replicates on UD-Q2_K_XL / llama.cpp / P100 sm_60. A turn that exits
to a tool carries a fraction of the reasoning that an answering turn does, on both stacks.

What it cannot do is discriminate your reading from mine, because my apparatus arm never fed a
tool result back — there is no turn 2 in that data. The turn-index gradient you propose is the
right discriminator and it needs the new arm. I am flagging that rather than letting a turn-1
number look like support it cannot provide.

Predictions, pre-registered

Re-reading both, our predicted outcomes agree more than the framing suggests. I said the
benefit would be "replaced by tool-call looping, which is the same non-termination wearing a
different bucket label"; you say task-level non-termination reappears "arriving as tool call
looping rather than think block looping." That is the same prediction.

The live disagreement is mechanism: I attributed it to the model failing to update on an
unhelpful result (belief), you attribute it to the exit truncating the reasoning episode before a
loop can develop (structure).

I am updating toward yours, because my own turn-1 data supports it and did not have to. If
episodes that exit to a call run ~203 tokens, a 16k cap is unreachable within one turn regardless
of what the model believes. Belief cannot be doing the work if the episode is cut before belief
matters.

My pre-registered predictions for the arm:

1. Per-turn cap hits stay near 1/492. Agreeing with you. High confidence.
2. Task-level non-termination reappears, landing within a factor of two of the no-apparatus
   arm's combined cap-hit + no-extractable rate (12 + 11 of 492). Agreeing with you.
3. Pre-call reasoning length does not grow materially with turn index — under 1.5× from turn
   1 to the last turn before exit. This is the one that discriminates, and it is the one I would
   have bet against a week ago. ~65% confidence, which is where I actually am rather than a
   number chosen to look decisive.
4. The majority of samples answer by turn 3 under fixed unhelpful returns. If this holds
   and task-level non-termination lands at or below baseline, you are falsified and routing is
   a real second fix. I put this at ~30%.

Protocol: the parts that are mine to pin

Agreed on 164 × K=3 = 492, max_tokens 16000, t0.7 / top_p 0.95 / top_k 20, turn cap 8,
bucketing by exit path, and reporting pooled and conditional with the attempting denominator
stated. The prompt and the three schemas are verbatim in the tarball — read_file(path),
write_file(path, content), run_command(command) — no paraphrase.

Fixed unhelpful returns, pinned. These are deliberately well-formed and unambiguous, so a model
that keeps calling is not doing so because the return was malformed:

read_file    -> {"ok": false, "error": "ENOENT: no such file or directory"}
run_command  -> {"ok": true, "exit_code": 0, "stdout": "", "stderr": ""}
write_file   -> {"ok": false, "error": "EROFS: read-only file system"}

Returned as the tool-result message content, verbatim, identical on both sides, on every call
regardless of arguments.

One addition I would like in the protocol: state cache_prompt / prefix-reuse configuration
and the concurrency setting in the result. Given the above, two runs of the same arm on the
same build are not necessarily comparable if those differ, and neither of us currently reports
them.

Happy to take the llama.cpp / P100 lane. Given your lanes are committed, I will likely have
numbers first — which is the argument for both sets of predictions being in the thread now,
above.

@apollo-mg this is the most useful thing anyone has handed me on this, thank you. Naming the three channels and pinning channel 2 to common/common.h:622 turns "our numbers wobble" into something I can actually test rather than worry about.

I ran your prefix check on our lane before replying, because if the flag we rely on does not do what we assume, I would rather find out from my own logs than from a reviewer later. Laguna S 2.1, llama.cpp Vulkan on gfx1151, single slot, temp 0, cache_prompt: false on every request:

cold  (slot flushed with an unrelated prompt, then X):    d2e3  d2e3  d2e3
warm  (slot warmed with X's own prefix, then X):          3392  3392  d2e3

Same request X, same flag, byte-hash of the output. The cold path is deterministic across three runs. The warm path diverges. So on our stack too, cache_prompt: false did not fully isolate an identical request from what was in the slot before it.

I want to be careful about what that does and does not show, because it is easy to over-read:

• It is an existence proof, not a rate. n=3 per arm. What it establishes is that non-isolation happens, not how often.
• I have not identified the mechanism. That warm[2] fell back to the cold hash means it is not even a clean deterministic flip, so I cannot honestly call this your channel 2 specifically versus some other slot-state carryover. I confirmed a phenomenon, not your mechanism.
• Most important: I measured bytes, and the thing I actually care about downstream is firing (does a think block appear at all). Your own framing is that cache order flips near-ties, and reasoning-present-versus-absent is not a near-tie. So I do not yet know whether this contamination can move a firing verdict or only the exact tokens. That is the check that would tell me whether it touches my results, and I have not run it.

Where this leaves my end, stated plainly so you can weight it:

• My robust finding (accumulated history collapses reasoning: 0/56 across shapes and seeds against baselines to 14/14) has a deterministic basis in the /apply-template render, so cache order does not reach it. I am confident in that one.
• My marginal cells (a 5/14 preserved-arm restoration on one prompt shape, p=0.041 uncorrected) are exactly where noise of this size could live, and I am no longer defending them. Your point retroactively supports the caution rather than creating a new problem.

Agreed on both of your protocol points, and I will adopt them: interleave, and state the cache flag explicitly in any protocol, since "we both ran it fresh" and "neither of us reused a prefix" are not the same claim. I would add a third from this: flushing the slot between samples is worth more than trusting the flag, at least on the stacks we can reproduce this on.

Your turn-1 scale replication is a clean result and I read it the same way you do: the structural premise holds on both stacks, and it cannot discriminate the readings, so the turn-index arm is the real discriminator. Looking forward to that run.

Drafted with AI assistance (Claude); reviewed and submitted by @Blackwellboy.

@Defilan running the check on your own lane before replying is the right order of operations, but the part worth singling out is what you did with the result. You kept the robust finding (accumulated history collapses reasoning, 0/56 across shapes and seeds against baselines at 14/14) because it has a deterministic basis in the /apply-template render, so cache order cannot reach it. You dropped the marginal cell (5/14 preserved-arm on one prompt shape, p=0.041 uncorrected) because that is exactly the size of effect this noise can manufacture. Same new evidence, two opposite verdicts, and what decides which is which is whether the claim has a mechanism the contamination cannot touch. That is the correct test and it is worth stating once, plainly, because the tempting move is to treat a newly discovered noise source as either fatal to everything or fatal to nothing.

You also named your open question precisely: you measured bytes, what you care about is firing, and you do not know whether cache contamination can move a behavioural verdict or only the exact tokens.

We can answer half of that from a third stack, because ours is not a byte difference.

What we measured

DeepSeek-V4-Flash (community-abliterated re-upload) on vLLM 0.21.1rc1.dev339+g1967a5627bc3, two DGX Spark GB10 nodes, tensor parallel 2, NVFP4 MLA KV cache at block size 256, sparse attention with a top-512 indexer, --max-num-batched-tokens 8192, --max-num-seqs 4, prefix caching and chunked prefill both enabled, multi-token-prediction drafter at depth 3. Temperature 0, greedy, max_tokens 32. A single fact planted in the first sentence of a long synthetic document, one question at the tail, prompts byte-identical within each pair. Cache hit fraction is prefix_cache_hits / prefix_cache_queries deltas taken around each individual request from the server's own metrics, so it is measured per request rather than assumed from a flag.

Ten requests, four documents, four prompt lengths:

Four runs at or below 79% cache coverage, four with finish_reason: length. Six runs at 99.8% or above, six with finish_reason: stop and a short exact answer. No overlap. The cold arm does not emit a stop token at all; it runs to the cap and continues generating invented document records. The warm arm answers in seven to nine tokens and stops.

Consistent with that, and reported separately because the cache fraction was not recorded on those requests: a cold depth ladder over six further documents at 31,999 to 131,068 tokens returned finish_reason: length on all six. Those are cold by time to first token (prefill at 853 to 969 tok/s) rather than by counter, so they support the direction and are not part of the contrast above.

This is a verdict flip, not a byte difference. stop versus length is not a near-tie in the sampler. The model stops answering the question and continues writing the document. So the answer to your open question, from a stack neither of you has touched, is that cache state can move a behavioural verdict and not only exact tokens.

What it does not show

It does not establish a mechanism. Our hypothesis is that a cold multi-chunk prefill and a cache replay compute the same KV by different kernel shapes over a quantised cache, and that a sparse top-512 indexer turns a small numeric difference into a different selected token set. That is a story, not a result. The decisive experiment is the same ladder with chunked prefill disabled, which needs a serve change we have not made.

It does not establish a rate. One cold run per length. The reason we are not leaning on any single pair is the same reason you flagged n=3: this lane is not deterministic at temperature 0, and that is measured rather than assumed. In a separate 30-request sweep at short context, six repeats each of five prompt families, prose, JSON and tool-calling reproduced byte-identically 6/6, a code prompt produced four distinct completion lengths in six runs (114, 107, 114, 157, 160, 160), and a maths prompt hit the 256-token cap five times and stopped cleanly at 244 once. Temperature-zero reproducibility here is task-dependent with an unproven cause. The MTP drafter is the obvious suspect and has not been isolated. What carries the claim above is the split across four lengths and four documents, not any one pair.

It does not show that contamination moved your result. Our failure mode is instruction collapse at long context. Yours is a near-tie flip at short context. Those are different phenomena, and the shared ingredient is only that server state decided the outcome. Treat ours as evidence that the class of failure exists, not as evidence about your cells.

One counterexample, because it is the only one we have and it cuts the other way. A single cold run at 131,068 tokens finished stop with the exact answer in 11 tokens. It is a framing control that places the question before the document, so its prefix diverges at token zero and it prefilled cold (157.07 s TTFT, against 1.4 to 1.8 s for the warm arms of the same control). It is not a member of the contrast above, but it says the cold path can reach the head of a 131k prompt and terminate cleanly when the instruction comes first. That points at instruction retention rather than at reach, and it is a live problem for our own mechanism paragraph.

Protocol, adopting your third point

Agreed and adopted: flushing the slot between samples is worth more than trusting the flag. We are folding that into the apparatus arm protocol pinned with @apollo-mg rather than leaving it as a note, because "we both set the flag" is now demonstrably not the same claim as "neither of us reused a prefix," and you established that on a lane where the flag was set explicitly.

The concrete consequence for the fed-back-tools arm, which is committed but not started: that arm is multi-turn and runs 492 samples sequentially against one server, so every sample inherits its predecessor's slot state, and its primary endpoints are exit-path buckets. Those are verdicts, not bytes. If cache state can move a verdict, the arm needs slot flushing between samples on both stacks, and the result should report four things rather than one: the prefix-reuse setting, the concurrency setting, the flush procedure used between samples, and at least one control demonstrating the flush actually isolates on that lane. Your cold arm is the shape that control should take. @apollo-mg, this is a change to the protocol we pinned, and it is cheaper to add now than to discover afterwards that the exit-path buckets carried slot state.

Credit where it converges

This independently strengthens trap 55 in @TheTom's open PR to our registry (a +21 to +24% prefill win that reproduced on a branch without the feature; warm caches and run order, about $15 of GPU time to disprove). His case was a performance number on his own hardware. Ours is a correctness and termination verdict on a stack he never measured, and yours and apollo's are two more. Worth noting that apollo's and yours are both llama.cpp family while ours is vLLM, so this is not a property of one serving layer.

We are staging a registry entry on the cold-versus-warm result. It will cite all three lines as independent convergence, each attributed by handle with its own conditions, and it will not merge anyone's numbers into ours or present a combined figure.

The cheap run that closes your half

The check that answers your remaining question on your own lane is the one you already built, scored differently: same cold-then-warm pair, but record whether a think block appears rather than the output hash. If firing is stable across the two arms while bytes move, contamination on your stack is confined to token identity, your robust finding was never at risk, and you withdrew the marginal cells more conservatively than you had to. If firing moves, you have what we have and the marginal cells stay withdrawn for the right reason. Either way it is two requests and it converts your "I have not run it" into a result.

Protocol change, flagged separately because you are about to start.

@apollo-mg this is in 5099808969 but buried, and it needs to reach you before the run rather than after.

Change to the protocol we already agreed, not a new request: flush the slot between samples, and run one cold control proving the flush actually isolates on that lane. Report the flush next to the flag and the concurrency setting, not instead of them.

The one line reason: your channel 2, plus @Defilan's check on a third stack where cache_prompt: false was set explicitly and the warm arm still diverged. Setting the flag is not evidence that nothing carried over. Your arm is 492 sequential samples against one server and its endpoints are exit path buckets, which are verdicts rather than bytes.

Full reasoning in 5099808969. Nothing else in the protocol changes.